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Highlights  of  GAO-12-727,  a  report  to 
congressional  committees 


AIR  FORCE  TRAINING 

Actions  Needed  to  Better  Manage  and  Determine 
Costs  of  Virtual  Training  Efforts 


Why  GAO  Did  This  Study 

Over  the  last  20  years,  the  Air  Force 
has  sought  ways  to  expand  its 
approaches  to  meeting  aircrew  training 
requirements,  including  the  increased 
use  of  virtual  training.  In  2012,  the  Air 
Force  reduced  live  flying  hours,  which 
it  estimates  will  save  $1.7  billion  in 
fiscal  years  2012  through  2016,  as  part 
of  its  response  to  the  Secretary  of 
Defense’s  efficiency  initiatives.  GAO 
conducted  this  study  in  response  to 
House  Report  112-78,  accompanying  a 
bill  for  the  Fiscal  Year  2012  National 
Defense  Authorization  Act,  which 
directed  GAO  to  review  the  status  of 
the  military  services’  virtual  training 
programs.  Specifically,  GAO  assessed 
(1 )  how  the  Air  Force  determines  the 
mix  of  live  and  virtual  training  to  meet 
training  requirements;  (2)  the  extent  to 
which  the  Air  Force  has  an  overarching 
organizational  framework  to  guide, 
oversee,  and  integrate  its  virtual 
training  efforts;  and  (3)  the  extent  to 
which  the  Air  Force  considered  costs 
related  to  virtual  training  in  estimating 
potential  savings  from  its  training 
efficiency  initiative.  To  do  so,  GAO 
analyzed  guidance  and  other 
documents,  visited  virtual  training 
facilities,  and  interviewed  officials  from 
the  Office  of  the  Secretary  of  Defense, 
the  Joint  Staff,  and  the  Air  Force. 

What  GAO  Recommends 

GAO  recommends  that  the  Air  Force 
designate  an  entity  to  integrate  its 
virtual  training  efforts,  develop  a 
strategy  to  align  virtual  training 
initiatives  and  goals,  and  develop  a 
methodology  to  collect  virtual  training 
cost  data.  DOD  concurred  with  GAO’s 
recommendations  and  identified 
planned  actions. 


View  GAO-12-727.  For  more  information, 
contact  Sharon  Pickup  at  (202)  512-9619  or 
pickups@gao.gov. 


What  GAO  Found 

The  three  lead  Air  Force  major  commands — Air  Mobility  Command,  Air  Force 
Special  Operations  Command,  and  Air  Combat  Command — all  utilize  training 
requirements  review  boards  composed  of  subject-matter  experts  to  determine 
training  requirements  for  specific  aircraft.  These  boards  determine  which  training 
requirements  can  be  completed  in  live  or  virtual  environments  based  upon 
factors  such  as  specific  combatant  command  mission  requirements  and  the 
capabilities  of  fielded  simulators  and  networks.  All  three  commands  use  a 
combination  of  live  and  virtual  approaches,  but  the  mix  varies  by  aircraft.  For 
example,  Air  Combat  Command  specifies  that  approximately  25  percent  of  its 
training  requirements  could  be  met  virtually.  The  other  two  commands  conduct 
approximately  50  percent  of  their  training  virtually. 

The  Air  Force  has  taken  steps  to  manage  its  virtual  training  efforts,  but  its 
approach  lacks  some  key  elements  of  an  overarching  organizational  framework 
needed  to  fully  integrate  efforts  and  address  challenges.  It  has  reorganized 
offices  and  undertaken  various  initiatives  intended  to  enhance  existing  virtual 
training  capabilities,  but  has  not  designated  an  entity  to  integrate  these  efforts  or 
developed  an  overarching  strategy  to  define  goals,  align  efforts,  and  establish 
investment  priorities.  As  a  result,  major  commands  have  developed  their  own 
investment  plans  and  standards  for  acquiring  and  fielding  virtual  training 
systems,  which  are  often  not  interoperable  and  require  costly,  time-consuming 
work-arounds  to  allow  personnel  to  train  together  and  with  joint  and  coalition 
partners.  GAO’s  prior  work  has  found  that  a  designated  entity  with  the  necessary 
authority  and  resources  and  an  overarching  strategy  are  critical  elements  of 
managing  organizational  transformations  and  meeting  long-term  goals  and 
agency  missions.  In  the  absence  of  an  approach  that  establishes  clear 
accountability  and  a  strategy  to  guide  its  planning  and  investment  decisions,  the 
Air  Force  will  continue  to  be  challenged  to  guide  the  efforts  of  its  commands  in 
planning  for  and  investing  in  virtual  training,  ensure  these  efforts  meet  the 
highest  priority  needs  and  are  synchronized  to  avoid  gaps  or  future 
interoperability  issues,  and  maximize  available  resources. 

The  Air  Force  estimated  it  could  save  about  $1.7  billion  in  its  training  program  by 
reducing  live  flying  hours  and  taking  other  steps,  such  as  increasing  the  use  of 
virtual  training,  but  it  lacks  a  methodology  for  determining  the  costs  of  virtual 
training  and  therefore  did  not  consider  these  costs  in  its  estimate.  The  Air  Force 
estimated  savings  based  solely  on  reductions  in  live  flying  hours  without 
considering  expenses  such  as  those  incurred  for  aircrew  to  travel  to  simulators, 
contractor  personnel  to  schedule  and  operate  simulators,  and  purchase  of 
additional  simulators.  GAO  has  found  that  decision  makers  need  visibility  over 
financial  data  to  meet  agency  goals  and  effectively  use  resources.  Identifying 
virtual  training  costs  is  challenging  because  data  is  spread  across  multiple 
program  elements  in  the  Air  Force’s  accounting  structure.  The  Air  Force 
completed  an  initial  study  in  September  201 1  that  identified  some  costs  related  to 
virtual  training,  but  it  concluded  these  data  might  not  be  complete.  In  the  absence 
of  taking  further  steps  to  determine  the  universe  of  costs  and  a  means  to  collect 
and  track  data,  the  Air  Force  will  be  limited  in  its  ability  to  make  fully  informed 
investment  decisions  about  the  mix  of  live  and  virtual  training  in  the  future. 
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Abbreviations 


DOD 

A30 

A30-CL 
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Department  of  Defense 

Air  Force  Director  of  Operations 

Air  Force  Director  of  Operations-Operational  Training 

Office  of  Security,  Counterintelligence,  and  Special 

Program  Oversight 

Program  Executive  Office-Simulation,  Training  and 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


July  19,  2012 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Howard  P.  “Buck”  McKeon 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

Over  the  past  two  decades,  the  Air  Force  has  sought  ways  to  move 
beyond  the  use  of  live  flying  to  meet  aircrew  training  requirements.  One 
alternative  it  has  pursued  is  to  increase  its  use  of  virtual  training,  which 
utilizes  simulators  and  information  networks  to  bring  together 
geographically  separated  units  in  order  to  conduct  training.1  Virtual 
training  allows  personnel  to  replicate  required  training  interactions  and 
procedures  while  reducing  fuel  costs  and  avoiding  other  constraints  that 
can  affect  live  training,  such  as  limited  access  to  ranges.  Air  Force 
leadership  considers  virtual  training  to  be  the  cornerstone  of  its  training 
transformation  efforts,  and  the  Air  Force  expects  to  rely  even  more 
heavily  on  virtual  training  as  it  fields  new  aircraft.  Further,  among  the 
initiatives  developed  in  response  to  the  Secretary  of  Defense’s  direction 
to  find  cost  savings  through  increased  department-wide  efficiency,  the  Air 
Force  reduced  its  fiscal  year  2012  budget  for  its  flying  hour  program, 
which  funds  live  training,  by  $268  million  while  calling  for  an  increase  in 
the  use  of  virtual  training. 

H.R.  Rep.  No.  112-78  (2011),  which  accompanied  a  bill  for  the  National 
Defense  Authorization  Act  for  Fiscal  Year  2012,  directed  us  to  review  the 
status  of  the  military  services’  training  programs  and  report  the  results  to 


Virtual  training  uses  aircraft  simulators  that  replicate  the  live  environment  for  aircrew 
missions  and  objectives.  Aircraft  simulators  can  be  stand-alone  or  linked  to  information 
networks.  Further  details  are  provided  below. 
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the  House  and  Senate  Armed  Services  Committees.2  For  this  review,  we 
assessed  (1)  how  the  Air  Force  determines  the  mix  of  live  and  virtual 
training  to  meet  training  requirements;  (2)  the  extent  to  which  the  Air 
Force  has  an  overarching  organizational  framework  to  guide,  oversee, 
and  integrate  its  virtual  training  efforts;  and  (3)  the  extent  to  which  the  Air 
Force  considered  costs  related  to  virtual  training  in  estimating  potential 
savings  from  its  training  efficiency  initiative.  We  are  reporting  separately 
on  the  other  services. 

To  address  these  objectives,  we  met  with  officials  from  the  Office  of  the 
Secretary  of  Defense,  Joint  Staff,  Office  of  the  Secretary  of  the  Air  Force, 
Headquarters  Air  Force,  and  several  Air  Force  major  commands,  and 
visited  virtual  training  facilities.  Our  review  focused  primarily  on  virtual 
training  systems  for  manned  aircraft  from  combat  air  forces,  mobility  air 
forces,  and  special  operations  forces.  To  determine  how  the  Air  Force 
determines  the  mix  of  live  and  virtual  training,  we  analyzed  Air  Force 
studies  and  assessments  of  virtual  training  technologies  and  capabilities. 
We  also  interviewed  officials  from  several  major  commands  and  obtained 
and  analyzed  training-requirement  instructions  for  combat,  mobility,  and 
special  operations  aircraft.  To  determine  the  extent  to  which  the  Air  Force 
has  developed  an  overarching  organizational  framework  to  guide, 
oversee,  and  integrate  its  virtual  training  efforts,  we  met  with  officials  from 
the  above  organizations  as  well  as  officials  from  Joint  and  Coalition 
Warfighting,  Joint  Training  Integration  and  Evaluation  Center,  Navy  Air 
Warfare  Center,  and  the  four  primary  centers  that  facilitate  distributed 
mission  operations.  To  determine  the  extent  to  which  the  Air  Force 
considered  costs  related  to  virtual  training  in  estimating  potential  savings 
from  its  training  efficiency  initiative,  we  obtained  and  analyzed  the  Air 
Force  efficiency  calculation  and  compared  it  with  cost-saving  estimating 
best  practices.  We  also  interviewed  officials  from  the  Air  Force  Defense 
Contracting  Management  Organization  and  Headquarters  Air  Force 
budget  office,  as  well  as  officials  from  the  major  commands  listed  above. 
More-detailed  information  about  our  scope  and  methodology  is  provided 
in  appendix  I. 

We  conducted  this  performance  audit  from  August  201 1  to  July  2012  in 
accordance  with  generally  accepted  government  auditing  standards. 


2H.R.  Rep.  No.  112-78  (2011),  which  accompanied  H.R.  1540,  a  bill  for  the  National 
Defense  Authorization  Act  for  Fiscal  Year  2012. 
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Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 


Background 


For  each  of  their  aircraft,  the  Air  Force  lead  commands  set  training 
requirements  that  aircrews  must  complete  on  an  annual  basis  in  order  to 
maintain  combat  mission  readiness.  These  training  requirements  include 
basic  tasks  such  as  take-offs  and  landings  and  also  more-advanced 
tasks,  such  as  air-to-air  combat  and  ground-attack  missions.  To  help 
meet  these  requirements,  the  Air  Force  has  developed  an  approach  to 
training  that  it  terms  distributed  mission  operations.  This  approach  is 
intended  to  train  units  as  they  expect  to  fight,  maintain  readiness,  and 
conduct  mission  rehearsals  in  a  realistic  environment.  Distributed  mission 
operations  utilizes  the  integration  of  virtual  (e.g.,  a  person  training  in  a 
simulator)  and  constructive  (e.g.,  computer  generated)  elements  to  train 
personnel  at  geographically  separated  sites  by  means  of  a  network.  For 
the  purposes  of  this  report,  we  refer  to  training  that  includes  a  simulator 
as  virtual  training. 

The  Air  Force  has  four  primary  centers  that  facilitate  distributed  mission 
operations  by  connecting  units  and  simulators  from  geographically 
dispersed  areas:3 

•  Distributed  Mission  Operations  Center  in  Albuquerque,  New  Mexico, 
managed  by  Air  Combat  Command; 

•  Distributed  Training  Operations  Center  in  Des  Moines,  Iowa, 
managed  by  the  Air  National  Guard; 

•  Warrior  Preparation  Center  in  Einsiedlerhof,  Germany,  managed  by 
U.S.  Air  Forces  Europe;  and 


3Air  Force  officials  stated  that  although  these  are  the  four  primary  centers,  there  are 
additional  centers  that  conduct  distributed  mission  operations.  For  example,  Air  Force 
Special  Operations  Command  manages  a  Mission  Rehearsal  Operations  Center  at 
Hurlburt  Field,  Florida. 
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•  Korean  Air  Simulation  Center  in  Osan,  Republic  of  Korea,  managed 
by  Pacific  Air  Forces. 

The  Distributed  Mission  Operations  Center  functions  as  the  lead 
integrator  of  virtual  systems  to  conduct  theater-level  exercises  and  events 
that  include  air,  land,  space,  cyber,  and  maritime  virtual  assets  for  Air 
Force,  joint,  and  coalition  partners.  These  large-scale  events,  known  as 
virtual  flags,  are  conducted  quarterly  and  last  about  2  weeks.  In  201 1 ,  the 
Distributed  Mission  Operations  Center  trained  over  1,400  personnel  from 
the  Air  Force,  Army,  Navy,  Marines,  and  coalition  forces  through  this 
virtual  exercise  as  well  as  other,  small-scale,  events. 

The  Distributed  Training  Operations  Center  plans,  builds,  and  manages 
small-scale  events  to  meet  the  learning  objectives  of  its  customers, 
mainly  Air  Combat  Command,  Air  National  Guard,  and  Air  Force  Reserve 
Command.  These  events  are  short-term,  typically  lasting  90  minutes. 
During  2011,  the  Distributed  Training  Operations  Center  conducted  over 
4,000  events  that  trained  more  than  9,500  personnel,  of  which  at  least  60 
percent  were  active-duty  personnel. 

The  two  overseas  virtual  training  centers  provide  different  capabilities  for 
the  commands  they  support.  The  Warrior  Preparation  Center  supports 
training  for  joint,  coalition,  and  partner-nation  forces  in  the  European  and 
African  theaters.  In  addition,  the  Warrior  Preparation  Center  supports  an 
Air-to-Ground  Operations  School  and  three  detachments  that  provide 
multinational  training  opportunities.  The  Korean  Air  Simulation  Center 
operates  constructive  simulations  that  support  the  air  operations  in  Korea 
during  U.S.  Forces  Korea  operational-level  exercises  and  supports 
selected  exercises  for  U.S.  Forces  Japan. 

To  train  its  units  and  personnel,  the  Air  Force  conducts  distributed 
mission  operations  using  several  different  internal  Air  Force  and 
Department  of  Defense  (DOD)  information  networks.  Some  of  these  key 
networks,  along  with  their  managing  organizations,  are  shown  in  table  1. 
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Table  1:  Virtual  Training  Networks 

Network 

Managed  by 

Distributed  Mission  Operations  Network 

Air  Combat  Command 

Air  Reserve  Component  Network 

Air  National  Guard 

Defense  Research  and  Engineering  Network 

High  Performance  Computing 
Modernization  Program 

Joint  Information  Operations  Range 

Deputy  Director  Joint  Force 
Development 

Joint  Training  Enterprise  Network 

Deputy  Director  Joint  Force 
Development 

Navy  Continuous  Training  Environment 

Navy 

Missile  Defense  Agency  Classified  Network 

Missile  Defense  Agency 

Point-to-Point  Connections 

Defense  Information  Systems  Agency 

Source:  Air  Combat  Command’s  Distributed  Missions  Operations  Center. 


These  networks  differ  according  to  such  factors  as  security  restrictions, 
bandwidth  capacity,  data  protocols,  and  support  services. 

In  May  2010,  the  Secretary  of  Defense  directed  DOD  to  undertake  a 
department-wide  efficiency  initiative  to  reduce  excess  overhead  costs  and 
to  reinvest  the  resulting  savings  in  sustaining  force  structure  and 
modernization.  The  Air  Force  identified  a  number  of  areas  to  improve  its 
efficiency,  including  an  initiative,  beginning  in  fiscal  year  2012,  to 
decrease  training  costs  by  reducing  its  live  flying  hour  program  for  its 
legacy  fighter  and  bomber  aircraft  by  5  percent  and  its  Air  Force  Reserve 
Command  F-16  flying  hour  program  by  10  percent.4  On  the  basis  of  this 
initiative,  the  Air  Force  expects  to  save  a  total  of  $1.7  billion  from  fiscal 
years  2012  to  2016.  The  Air  Force  estimated  savings  of  about  $268 
million  for  fiscal  year  2012.  In  discussing  the  initiative,  the  Air  Force 
stated  that  it  expected  to  offset  any  effect  on  readiness  caused  by  a 
reduction  in  live  flying  hours  by  increasing  its  use  of  simulators. 


4Legacy  fighter  aircraft  are  the  F-1 5,  F-1 6,  and  A-1 0,  and  legacy  bomber  aircraft  are  the 
B-1  and  B-52. 
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Air  Force  Major 
Commands  Determine 
Their  Mix  of  Live  and 
Virtual  Training  Based 
on  Various  Factors 


Currently,  the  three  lead  Air  Force  major  commands — Air  Mobility 
Command,  Air  Force  Special  Operations  Command,  and  Air  Combat 
Command — have  similar  processes  to  determine  the  mix  of  live  and 
virtual  training,  but  the  mix  of  training  differs  across  the  major  commands, 
and  among  aircraft  within  the  commands.  Air  Combat  Command  is 
responsible  for  fighters,  bombers,  and  attack  aircraft;  Air  Mobility 
Command  is  responsible  for  transport  and  tanker  aircraft;  and  Air  Force 
Special  Operations  Command  is  responsible  for  special-operations 
aircraft.  At  each  command,  training-requirement  review  boards  composed 
of  subject-matter  experts  meet  to  consider  broad  sets  of  training  issues 
and  evaluate  training  requirements  for  specific  aircraft.5  The  boards 
consider  factors  such  as  specific  combatant  command  mission 
requirements  and  the  capabilities  of  simulators  and  networks  that  have 
already  been  fielded,  and  determine  which  training  requirements  can  be 
completed  in  a  virtual  environment  and  which  need  to  be  completed  in  a 
live  environment.  The  results  of  their  reviews  are  reflected  in  updated 
training  guidance  for  each  type  of  aircraft.  In  addition,  each  of  the 
commands  also  establishes  requirements  to  improve,  acquire,  or  upgrade 
training  devices  to  meet  mission  tasks. 


While  all  three  lead  major  commands  rely  on  both  live  and  virtual  training 
to  meet  aircrews’  training  requirements,  the  mix  is  different  for  each  major 
command,  as  discussed  below. 


Air  Mobility  Command  For  each  aircraft  type,  Air  Mobility  Command  issues  a  requirement 

document  that  specifies  the  number  of  times  each  task  or  “event”  must  be 
completed  for  a  pilot  or  aircrew  to  be  certified  as  mission  ready.  The 
document  also  specifies  the  percentage  of  events  that  can  be  completed 
in  a  simulator.  For  example,  the  C-130  requirement  document  specifies 
that  50  percent  of  assault  landings  may  be  completed  in  a  simulator  and 
100  percent  of  instrument  approaches  in  a  simulator.  Although  live  and 
virtual  training  requirements  vary  by  aircraft,  according  to  Air  Mobility 
Command  officials,  approximately  50  percent  of  aircrew  training  is 
conducted  in  simulators,  including  all  training  related  to  takeoffs,  landings, 


5Air  Mobility  Command  and  Air  Combat  Command  refer  to  their  training-requirement 
review  boards  as  “Realistic  Training  Review  Boards”  while  Air  Force  Special  Operations 
Command  refers  to  its  board  as  a  “Simulator  and  Training  System  Integrated  Product 
Team.” 
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and  instrument  approaches.6  Due  to  limitations  in  simulator  fidelity, 
however,  training  for  some  special  qualifications  such  as  aerial  refueling, 
formation  flying,  airdrops,  and  assault  landings  must  periodically  be 
conducted  live  in  the  actual  aircraft.  For  example,  for  aerial  refueling, 
currently,  there  are  differences  between  what  the  fighter  pilots  see  in  their 
simulators  and  what  air  refueling  crews  see  in  their  simulators.  Because 
the  simulators  are  currently  not  able  to  accurately  replicate  the  aerial 
refueling  environment,  simulated  training  cannot  yet  replace  live  training. 
In  developing  its  virtual  training  program,  Air  Mobility  Command  worked 
with  the  Federal  Aviation  Administration  to  leverage  civilian  standards, 
which  require  simulators  to  respond  like  the  actual  aircraft  in  order  to  be 
certified  for  training.  Air  Mobility  Command  is  currently  developing  a 
networked  distributed  training  center  that  would  enable  more  virtual 
training  with  combat  air  forces  and  coalition  partners. 


Air  Force  Special  As  Air  Mobility  Command  does,  for  each  aircraft  type  Air  Force  Special 

Operations  Command  Operations  Command  issues  a  requirement  document  that  specifies  the 

number  of  times  each  task  or  “event”  must  be  completed  for  a  pilot  or 
aircrew  to  be  certified  as  mission  ready  and  the  percentage  of  events  that 
can  be  completed  in  a  simulator.  Air  Force  Special  Operations  Command 
officials  stated  that  the  command’s  goal  is  to  accomplish  up  to  50  percent 
of  its  aircrew  training  in  simulators  depending  upon  the  aircraft.  For 
example,  aircrew  training  requirements  for  the  AC-130U,  a  close  air 
support  aircraft,  allows  aircrew  to  accomplish  50  percent  of  their  mission 
tasks  in  a  simulator.  Air  Force  Special  Operations  Command  based  its 
simulator  certification  program  on  the  standards  and  metrics  used  by  the 
Federal  Aviation  Administration  and  Air  Mobility  Command.  Air  Force 
Special  Operations  Command  officials  stated  that  simulators  provide 
training  that  might  not  be  available  in  the  live  environment,  such  as 
training  for  specific  locations  or  adverse  weather  conditions.  Air  Force 
Special  Operations  Command  has  a  stated  goal  to  perform  all 
qualification  and  continuation  training  events  in  the  simulator,  while 
increasing  both  live  and  simulator  mission  rehearsal  training. 


6Officials  noted  that  percentages  can  vary  due  not  only  to  differences  in  aircraft  and 
simulators  but  also  depending  on  whether  the  percentages  are  based  on  requirements  or 
events.  Some  complex  requirements  might  only  have  to  be  performed  once  or  twice 
during  each  training  cycle  while  other  events  such  as  takeoffs  and  landings  have  to  be 
performed  more  regularly. 
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Air  Combat  Command 


Air  Combat  Command  also  issues  a  requirement  document  for  each  type 
of  aircraft  on  an  annual  basis.  Virtual  training  requirements  vary  by 
aircraft,  with  large  aircraft  such  as  bombers  generally  able  to  satisfy  more 
of  their  training  requirements  in  simulators  than  fighters.  Beginning  in 
fiscal  year  2012,  Air  Combat  Command’s  training-requirement  review 
board  revised  each  aircraft’s  training  requirements  and  specified  that 
approximately  25  percent  of  training  requirements  were  to  be  met  using 
virtual  training,  while  the  rest  of  the  requirements  were  to  be  met  using 
live  training.  Prior  to  this,  Air  Combat  Command  training  guidance 
specified  that  virtual  training  was  to  be  used  as  a  supplement  to  live 
training,  but  it  did  not  set  a  goal  or  specific  percentage  requirement  for 
virtual  training.  The  virtual  training  that  had  been  done  in  those  years 
included  emergency  procedures,  instrumentation  training,  and  tactical 
training  rather  than  mission  training.  With  the  availability  of  more- 
advanced  full  mission-training  simulators,  aircrews  are  now  able  to  train 
beyond  these  basic  tasks  to  more-advanced  air-to-air  and  air-to-ground 
combat  missions,  like  suppression  of  enemy  air  defenses. 

According  to  Air  Combat  Command  officials,  the  combat  air  forces  face 
certain  challenges  that  prevent  them  from  conducting  the  same  level  of 
virtual  training  as  forces  from  the  other  major  commands.  Some 
challenges  arise  due  to  differences  between  unit  and  simulator  locations, 
difficulties  coordinating  distributed  training  events,  and  a  lack  of  simulator 
fidelity.  For  example,  officials  stated  that  there  are  very  few  simulators 
collocated  with  Reserve  component  units,  which  means  valuable  reserve 
component  training  time  can  be  lost  travelling  to  and  from  the  simulators. 
Officials  also  noted  that  the  software  for  some  aircraft  simulators  is  two  or 
three  versions  behind  the  software  in  the  actual  aircraft,  which  could  in 
some  cases,  affect  the  performance  of  aircrews  in  the  actual  aircraft.  In 
addition,  fighter  simulators  cannot  replicate  the  extreme  physical  effects 
of  air  combat  maneuvers  that  fighter  pilots  experience  in  the  actual 
aircraft.  Fighter  pilots  we  interviewed  stated  that  unlike  flying  other  aircraft 
such  as  bombers  and  transports,  fighter  pilots  must  effectively  make 
decisions  while  conducting  their  missions  in  a  hostile  environment  and 
maneuvering  the  aircraft  through  high-speed  and  high-gravity  maneuvers 
that  put  stress  on  the  human  body.  We  note  that  the  Navy  faces  similar 
challenges  in  conducting  virtual  training  for  its  fighter  aircrews.  For 
example,  the  crews  of  the  Navy’s  F/A-18E/F  currently  conduct  18  percent 
of  their  training  through  virtual  training  and  plan  to  increase  this  to  32 
percent  by  2020. 
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The  Air  Force  Has 
Not  Established  an 
Overarching 
Organizational 
Framework  to  Guide, 
Oversee,  and 
Integrate  Virtual 
Training  Efforts 


The  Air  Force  has  recently  taken  steps  to  increase  management  attention 
over  its  virtual  training  efforts,  but  its  approach  to  virtual  training  currently 
lacks  (1)  a  designated  organization  with  accountability  and  authority  for 
achieving  results  and  (2)  an  overarching  strategy — key  elements  of  an 
organizational  framework  that  we  have  found  to  be  critical  for  successful 
transformations  in  both  public  and  private  organizations.7  In  the  absence 
of  a  framework  to  structure  and  guide  its  virtual  training  efforts,  the  Air 
Force  will  continue  to  face  challenges  in  integrating  its  virtual  capabilities 
and  cannot  be  certain  that  its  efforts  align  with  strategic  goals  or  know 
whether  critical  gaps  or  duplication  of  efforts  exist. 


The  Air  Force  Has 
Increased  Management 
Focus  on  Virtual  Training 
Efforts,  but  Oversight 
Remains  Fragmented 


According  to  Air  Force  leadership,  distributed  mission  operations  are  the 
cornerstone  of  the  Air  Force  training  transformation.  Additionally,  in  the 
Strategic  Plan  for  the  Next  Generation  of  Training,  DOD  has  emphasized 
comprehensive  training  that  integrates  service  and  joint  capabilities.8  The 
Air  Force  has  increased  management  attention  on  virtual  training  efforts 
by  reorganizing  and  creating  new  headquarters  offices  and  establishing 
working  groups,  but  oversight  remains  fragmented.  For  instance,  in 
February  201 1 ,  the  Air  Force  Agency  for  Modeling  and  Simulation  was 
realigned  under  Headquarters  Air  Force  Director  of  Operations  (A30)  to 
serve  as  the  execution  arm  for  integrating  and  implementing  virtual 
capabilities,  resources,  and  policy.  Also,  in  August  201 1 ,  the  Air  Force 
established  the  Headquarters  Air  Force  Director  of  Operations- 
Operational  Training  (A30-CL)  office  to  provide  leadership  and  support  to 
distributed  mission  operations  users  across  the  Air  Force.  Additionally,  in 
February  2012,  Headquarters  Air  Force  established  a  working  group, 
composed  of  subject-matter  experts  from  the  Distributed  Training  Centers 
and  the  major  commands,  to  address  operational  challenges  within  the 
virtual  training  programs.  Issues  unable  to  be  resolved  in  this  forum  are 
elevated  to  higher-level  working  groups  including  the  Headquarters 
Operations  Conference,  the  Modeling  and  Simulation  Steering 


7 GAO,  Results-Oriented  Cultures:  Implementation  Steps  to  Assist  Mergers  and 
Organizational  Transformations,  GAO-03-669  (Washington,  D.C.:  July  2,  2003). 

8Office  of  the  Under  Secretary  of  Defense,  Personnel  &  Readiness,  Readiness  and 
Training  Policy  and  Programs,  Strategic  Plan  for  the  Next  Generation  of  Training  for  the 
Department  of  Defense  (Sept.  23,  2010). 


Page  9 


GAO-12-727  Air  Force  Training 


Committee,  and  finally  the  Air  Force  Modeling  and  Simulation  General 
Officer’s  Steering  Group. 

These  organizations  and  working  groups  have  increased  management 
focus  on  virtual  training  efforts,  but  the  Air  Force  has  not  designated  an 
organization  with  accountability  and  oversight  authority  necessary  to 
integrate  all  its  virtual  training  efforts,  including  developing  and  acquiring 
interoperable  virtual  training  systems  and  establishing  and  enforcing 
authoritative  standards  for  simulators,  constructive  elements,  and 
databases.  Rather,  oversight  of  standards  development,  acquisition, 
sustainment,  and  integration  of  training  systems  is  fragmented  among 
various  Air  Force  organizations,  as  shown  in  table  2. 


Table  2:  Air  Force  Virtual  Training  Roles 

Office 

Roles 

Headquarters  Air  Force  Director  of  Operations- 
Modeling  and  Simulation  (A30-CM) 

Provide  modeling  and  simulation  policy,  governance,  and  processes  to  achieve  full 
benefit  of  modeling  and  simulation  across  Air  Force. 

Headquarters  Air  Force  Director  of  Operations- 
LVC-Operational  Training  (A30-CL) 

Provide  leadership  and  support,  and  ensure  capabilities  are  available  for  live,  virtual, 
and  constructive  operational  training  and  distributed  mission  operations. 

Air  Force  Agency  for  Modeling  and  Simulation 

Provide  implementation  and  support  of  Air  Force,  Joint,  and  DOD  modeling  and 
simulation  policy  and  standards.3 

Air  Force  lead  major  commands 

Establish  standards,  tasks,  and  formal  training  requirements  for  training  systems, 
including  systems  required  for  distributed  mission  operation  capabilities. 

Air  Force  Materiel  Command 

Execute  research,  development,  test  and  evaluation,  and  oversee  distributed 
mission  operations  acquisition,  sustainment,  and  support. 

Air  Force  Research  Laboratory-Warfighter 
Readiness  Research  Division 

Execute  the  research  and  development  for  warfighter  training  systems. 

Aeronautical  Systems  Center-Simulators 
Division 

Execute  the  acquisition,  development,  and  sustainment  of  aircraft  simulators. 

Electronic  Systems  Center 

Develop  and  acquire  systems  that  combine  computers,  radars,  information  displays, 
and  communication  gear. 

Air  Combat  Command  Distributed  Mission 
Operations  Center 

Serve  as  the  lead  integrator  for  theater-level  distributed  mission  operations  for  the 

Air  Force,  joint  forces,  and  coalition  partners. 

Office  of  Security,  Counterintelligence,  and 
Special  Program  Oversight  (SAF/AAZ) 

Serve  as  accrediting  authority  for  distributed  mission  operation  events  and  exercises 
to  ensure  compliance  with  special-access  program  policies  and  procedures. 

Source:  GAO. 


The  Air  Force  Agency  for  Modeling  and  Simulation  mission  directive  is  currently  being  revised  to 
include  its  reassignment  as  an  organizational  element  of  Headquarters  Air  Force,  with  updated  roles 
and  responsibilities. 
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In  the  absence  of  an  organization  to  guide  virtual  training  efforts,  the  lead 
major  commands  have  developed  their  own  standards  and  acquired  and 
fielded  systems  that  are  not  interoperable  and  often  require  costly,  time- 
consuming  work-arounds  to  be  able  to  train  together  in  large,  complex 
virtual  training  exercises.  For  example,  in  conducting  its  quarterly  virtual 
flag  exercises,  the  Distributed  Mission  Operations  Center  must  integrate 
simulators  and  networks  that  have  been  developed  to  different  standards. 
To  integrate  all  the  participants  into  the  exercises,  the  center  must 
implement  “gateways”  that  allow  dissimilar  simulators  to  translate  data; 
develop  databases  to  provide  a  common  constructive  environment;  and 
link  numerous  Air  Force  and  DOD  networks  that  have  different  security 
restrictions,  bandwidth  limitations,  and  data  transfer  protocols.9  According 
to  officials,  developing  and  implementing  these  types  of  solutions  takes 
up  to  9  months,  and  involves  significant  reliance  on  contractor  personnel. 
The  network  configuration  must  be  certified  for  each  virtual  flag  event  and 
then  it  must  be  disassembled,  reconfigured,  and  recertified  for 
subsequent  training  events.  While  the  Air  Force  has  been  developing 
these  work-around  solutions  to  allow  interoperability  among  its  older 
aircraft  simulators,  similar  solutions  will  be  also  be  needed  for  Joint  Strike 
Fighter  simulators  as  they  are  fielded  and  integrated  into  distributed 
mission  operations.  According  to  Joint  Strike  Fighter  program  office 
officials,  the  programs’  operational  requirement  document  specifies  that 
the  aircraft  and  simulators  must  be  interoperable  with  other  aircraft  and 
networks,  but  interoperability  is  not  scheduled  to  be  achieved  until  later 
blocks  in  the  program’s  development.  The  Air  Force’s  current  Joint  Strike 
Fighter  simulators  are  stand-alone  and  not  integrated  into  distributed 
mission  operations.  Like  other  aircraft  simulators,  the  Joint  Strike  Fighter 
simulators  will  require  complex  multilevel  security  guards  and  gateways  to 
allow  them  to  operate  with  other  aircraft  simulators  in  a  distributed 
mission  environment.10  Joint  Strike  Fighter  program  officials  stated  that 
since  the  types  of  interoperability  challenges  that  they  face  are  not  unique 
but  are  similar  to  those  of  other  programs  there  should  not  be  any  unique 
technical  barriers  that  would  prevent  solutions  to  these  challenges.  Air 


9Gateways  are  information-technology  solutions  that  translate  data  between  simulators 
that  were  developed  using  different  simulation  standards. 

i  n 

A  multilevel  security  guard  is  a  computer  software  program  that  allows  a  system  to 
process  information  with  different  security  classifications,  permits  access  to  users  with 
different  security  clearances,  and  prevents  users  from  obtaining  access  to  information  for 
which  they  lack  authorization. 
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Combat  Command  is  also  currently  assessing  the  technologies  needed  to 
integrate  live  and  virtual  training  for  the  Joint  Strike  Fighter. 

Our  prior  work  has  found  that  designating  an  integration  team,  vested 
with  the  necessary  authority  and  resources,  is  a  critical  element  of 
managing  the  transformation  of  an  organization.11  We  note  that  the  Army 
and  the  Navy  have  each  designated  an  organization  with  centralized 
oversight  over  standards  development,  acquisition,  sustainment,  and 
integration  of  virtual  training  systems.12  Further,  the  Navy  has  established 
guiding  principles  and  investment  priorities  to  assist  decision  makers  in 
selecting  the  proper  simulator  solution  for  specific  training  requirements 
and  gaps,  and  to  help  avoid  interoperability  issues.  One  of  the  principles 
states  that  simulators  intended  to  interface  with  other  simulators  must  be 
compatible  with  the  Navy  Continuous  Training  Environment  network.  A 
Navy  training  instruction  elaborates  on  this  guiding  principle  by  further 
stating,  “Interoperability  is  a  key  objective  for  Fleet  simulators.  Translator 
development  is  expensive  and  time  consuming,  translation  slows  things 
down  (introduces  latency),  and  translation  is  never  perfect.  To  reduce  the 
need  for  translators  to  overcome  interoperability  challenges,  Navy 
Continuous  Training  Environment  network  technical  standards  have  been 
adopted  Fleet-wide  and  are  mandatory  for  new  simulators  that  will 
integrate  into  Fleet  synthetic  training.” 

DOD  has  called  for  an  approach  to  training  investments  that  eliminates 
“after-the-fact”  interoperability  solutions  for  training  capabilities  and  the 
breaking  down  of  stovepipes  to  achieve  an  efficient  yet  realistic  training 
environment.13  Also,  in  201 1  the  Air  Force  Agency  for  Modeling  and 
Simulation  developed  an  Initial  Capabilities  Document  that  has  been 
approved  by  the  Joint  Requirements  Oversight  Council  and  that  called  for 
an  organization  with  contracting  oversight  that  could  provide  specific 
requirements  supporting  improved  interoperability  across  the  virtual 


^GAO,  Results-Oriented  Cultures:  Implementation  Steps  to  Assist  Mergers  and 
Organizational  Transformations,  GAO-03-669  (Washington,  D.C.:  July  2,  2003). 

12The  Program  Executive  Office — Simulation,  Training  and  Instrumentation  (PEO-STRI)  is 
the  acquisition  and  contracting  center  for  live,  virtual,  and  constructive  training  aids, 
devices,  simulations  and  simulators  for  the  Army.  The  Naval  Air  Warfare  Center  is  the 
principal  Navy  center  for  research,  development,  test  and  evaluation,  acquisition  and 
product  support  of  training  systems. 

13Office  of  the  Under  Secretary  of  Defense,  Strategic  Plan  for  the  Next  Generation  of 
Training  for  Department  of  Defense  (Sept.  23,  2010). 
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training  environment.14  However,  the  Air  Force  has  not  yet  identified  an 
organization  to  perform  this  oversight.  Without  a  dedicated  organization 
with  responsibility  and  accountability  to  integrate  virtual  training  efforts, 
the  Air  Force  may  continue  to  face  challenges  in  managing  and 
integrating  its  virtual  training  efforts,  including  interoperability  issues  that 
lead  to  diminished  training  quality,  fewer  training  opportunities  due  to 
lengthy  preparation  times,  and  increased  costs. 


Individual  Air  Force 
Initiatives  to  Enhance 
Virtual  Training 
Capabilities  Are  Not  Yet 
Guided  by  an  Overarching 
Strategy 


•  The  Distributed  Training  Operations  Center  has  increased  its 
capabilities  in  response  to  increased  user  requirements  and  mission 
requests  by  adding  personnel  and  expanding  event  availability. 
Additionally,  the  Distributed  Training  Operations  Center  plans  to  work 
with  the  major  commands  to  establish  remote  capability  sites  that 
would  be  networked  to  the  center  to  increase  distributed  mission 
operations  availability  across  the  Air  Force.  According  to  Distributed 
Training  Operations  Center  officials,  remote-capability  sites  have 
already  been  established  for  the  Air  National  Guard  in  South  Dakota 
and  Pacific  Air  Forces  in  Alaska. 

•  Air  Combat  Command  is  in  the  process  of  establishing  a  Distributed 
Training  Center  at  Langley  Air  Force  Base.  The  training  center  is 
intended  to  provide  a  focal  point  for  scheduling  of  combat  air  forces 


The  Air  Force  is  currently  pursuing  a  number  of  individual  initiatives  to 
enhance  its  virtual  training  capabilities.  Among  these  initiatives  are  the 
following: 

•  Air  Mobility  Command  is  planning  to  create  and  staff  a  Distributed 
Training  Center  at  Scott  Air  Force  Base  in  December  2012.  The 
training  center  is  initially  planned  to  provide  the  integration  capability 
needed  for  transport  aircraft  simulators  to  conduct  distributed  mission 
operations  on  a  daily,  consistent  basis  and  will  also  provide  the  ability 
for  these  simulators  to  participate  in  Distributed  Mission  Operations 
Center  events.  Air  Mobility  Command  also  has  plans  to  expand  the 
center’s  capability  to  include  integration  of  tanker  simulators  to  train 
air  refueling  virtually. 


1 4 

United  States  Air  Force  Agency  for  Modeling  and  Simulation,  Initial  Capabilities  Report 
for  Integrating  Architecture  for  Air  &  Space  Live ,  Virtual ,  and  Constructive  Environments 
(IA-ASLVCE)  (Sept.  12,  2011). 
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events  not  involving  the  Distributed  Training  Operations  Center  or 
Distributed  Mission  Operations  Center.  It  is  also  intended  to  provide 
scenario  development,  focused  on  training  gaps,  desired  unit 
missions,  operational  plan  missions,  and  other  tactics,  techniques, 
and  procedures.  The  training  center  is  expected  to  be  fully  operational 
in  September  2012. 

•  Air  Force  Special  Operations  Command  is  in  the  process  of 

establishing  a  Distributed  Training  Center  at  Cannon  Air  Force  Base 
that  will  focus  on  virtual  training  activities  for  initial  and  mission 
qualification  training  and  unit-level  mission  essential  tasks. 

Our  prior  work  has  found  that  strategic  planning  is  a  key  element  of  an 
overarching  organizational  framework.  For  example,  a  leading  practice 
derived  from  principles  established  under  the  Government  Performance 
and  Results  Act  of  1 993  is  to  improve  the  management  of  federal 
agencies  by  developing  comprehensive  strategies  to  address 
management  challenges  that  threaten  their  ability  to  meet  long-term 
goals.15  We  have  previously  reported  that  these  types  of  strategies  should 
contain  results-oriented  goals,  performance  measures,  and  expectations 
with  clear  linkages  to  organizational,  unit,  and  individual  performance 
goals  to  promote  accountability  and  should  also  be  clearly  linked  to  key 
resource  decisions.16  While  the  Air  Force  currently  has  numerous 
individual  initiatives  underway  to  enhance  its  virtual  training  capabilities 
and  is  planning  to  make  additional  investments,  it  has  not  yet  developed 
an  overall  strategy  to  guide  and  integrate  these  efforts.  For  example,  the 
Air  Force  has  not  outlined  overall  goals  for  its  virtual  training  efforts, 
resource  needs,  and  investment  priorities.  In  the  absence  of  a  strategy, 
the  Air  Force  cannot  be  certain  that  its  individual  initiatives  are 
synchronized  and  will  address  its  highest  priority  needs. 


15Pub.  L.  No.  103-62  (1993),  as  amended  by  Pub.  L.  No.  111-352  (2011). 

16See,  for  example,  GAO,  Highlights  of  a  GAO  Roundtable:  The  Chief  Operating  Officer 
Concept:  A  Potential  Strategy  to  Address  Federal  Governance  Challenges, 
GAO-03-192SP  (Washington,  D.C.:  Oct.  4,  2002);  Highlights  of  a  GAO  Forum:  Mergers 
and  Transformation:  Lessons  Learned  for  a  Department  of  Homeland  Security  and  Other 
Federal  Agencies,  GAO-03-293SP  (Washington,  D.C.:  Nov.  14,  2002);  Defense  Business 
Transformation:  Achieving  Success  Requires  a  Chief  Management  Officer  to  Provide 
Focus  and  Sustained  Leadership,  GAO-07-1072  (Washington,  D.C.:  Sept.  5,  2007);  and 
Unmanned  Aircraft  Systems:  Additional  Actions  Needed  to  Improve  Management  and 
Integration  of  DOD  Efforts  to  Support  Warfighter  Needs,  GAO-09-175  (Washington,  D.C.: 
Nov.  14,  2008). 
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Air  Force  officials  stated  that  they  are  currently  developing  a  Live,  Virtual, 
Constructive  Flight  Plan  to  serve  as  their  strategy  for  virtual  training.  They 
told  us  the  Flight  Plan  will  provide  direction  to  the  major  commands  on  the 
handling  of  operational  issues  and  will  establish  an  internal  structure  for 
how  issues  are  to  be  raised  and  resolved.  Officials  expect  that  the  Flight 
Plan  will  be  completed  by  July  2012.  Officials  stated  that  a  separate  effort 
will  be  undertaken  to  develop  an  acquisition  strategy  for  virtual  training 
systems.  At  this  point,  it  is  unclear  the  extent  to  which  these  plans  will 
contain  the  necessary  elements  of  an  overall  strategy  that  the  Air  Force 
can  use  to  manage  and  integrate  its  planning  and  acquisition  efforts. 


Air  Force  Lacks  Cost 
Data  for  Virtual 
Training  and  Did  Not 
Account  for  Virtual 
Training  Costs  When 
Determining  Savings 
from  Its  Flying-Hour 
Efficiency  Initiative 


In  outlining  its  efficiency  initiative  related  to  training,  the  Air  Force 
estimated  potential  cost  savings  of  $268  million  for  fiscal  year  2012,  and  a 
total  of  $1.7  billion  for  fiscal  years  2012  to  2016  by  among  other  things, 
reducing  legacy  combat  Air  Force  flying  hours  across  the  board  by  5 
percent.17  The  flying-hour  efficiency  initiative  also  called  for  a  concurrent 
increase  in  the  use  of  high-fidelity  simulators  and  virtual  training  to  avoid 
any  effect  on  aircrews’  mission  readiness  from  the  reduction  in  live  flying. 
However,  in  estimating  costs,  the  Air  Force  included  the  savings 
associated  with  reductions  in  live  training  but  not  the  potential  costs 
associated  with  increases  in  virtual  training  that  were  called  for  to  offset 
the  reduction  in  live  training.  On  the  basis  of  our  prior  work,  cost  savings 
estimates  should  include  all  significant  costs  in  order  to  have  a 
reasonable  basis.  Additionally,  federal  internal  control  standards  state 
that  decision  makers  need  visibility  over  a  program’s  financial  data  to 
determine  whether  the  program  is  meeting  the  agencies’  goals  and 
effectively  using  resources.18 

Air  Force  officials  told  us  that  the  cost  savings  associated  with  the  flying- 
hour  efficiency  initiative  were  estimated  by  multiplying  the  reductions  in 
live  training  flying  hours  for  each  aircraft  by  the  cost  per  flying  hour  for 
that  aircraft,  and  then  adding  the  resultant  figures  for  all  the  aircraft  to 
determine  total  savings.  For  example,  according  to  numbers  provided  by 
the  Air  Force,  the  live  training  cost  of  1  F-15E  flight  hour  is  approximately 


17Department  of  Defense,  Department  of  Defense  Efficiency  Initiatives,  Fiscal  Year  2012 
Budget  Estimates  (Feb.  18,  2011). 

18GAO,  Standards  for  Internal  Control  in  the  Federal  Government,  GAO/AIMD-OO-21.3.1 
(Washington,  D.C.:  Nov.  1,  1999) 
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$17,449  and  F-15E  flying  hours  were  reduced  by  1,782  hours.  These 
amounts  were  multiplied  together  to  arrive  at  the  Air  Force’s  projected 
total  savings  of  approximately  $31 ,094,000  for  reductions  in  F-1 5E  flight 
hours.  Similar  calculations  were  made  for  each  of  the  other  aircraft  that 
had  their  flight  hours  reduced  and  the  savings  for  all  the  aircraft  were 
summed. 

The  Air  Force  did  not  consider  any  potential  costs  associated  with  the 
increase  in  virtual  training  in  its  estimate  of  cost  savings  because  it  has 
not  developed  a  methodology  to  collect  and  track  information  on  the  cost 
of  its  virtual  training  program.  According  to  Air  Force  officials,  some 
training  costs  could  increase  as  a  result  of  increases  in  virtual  training. 
These  costs  could  include  expenses  for  aircrew  to  travel  to  simulator 
locations,  additional  contractor  personnel  to  schedule  and  operate 
simulators,  and  the  purchase  of  additional  simulators  to  meet  increased 
demand.  Furthermore,  according  to  Air  Force  officials,  identifying  virtual 
training  costs  is  challenging  because  funds  to  support  virtual  training  and 
distributed  mission  operations  are  currently  dispersed  across  multiple 
program  elements.19  For  example,  our  analysis  identified  a  portion  of 
virtual  training  funding,  specifically  distributed  mission  operations  funding, 
in  a  program  element  titled  “Human  Effectiveness  Applied  Research.”  In 
another  case,  distributed  mission  operations  funding  was  part  of  a 
program  element  titled  “International  Activities,”  under  an  “Armaments 
Cooperation”  subcategory  that  also  included  funding  for  alternative 
energy  among  other  things.  In  201 1 ,  the  Air  Force  conducted  a  onetime 
study  in  an  attempt  to  identify  the  full  cost  of  its  virtual  efforts.20  It  found 
that  the  total  investment  in  virtual  capabilities  for  fiscal  year  2012  was  at 
least  $1.9  billion.  Of  that,  operational  training  support  accounted  for 
approximately  50  percent  of  the  annual  investment,  including  the  largest 
identified  expenditure  of  $1 82.3  million  for  combat  air  forces  distributed 
mission  operations.  However,  the  study  noted  that  its  efforts  may  not 
have  identified  all  program  elements  associated  with  virtual  training  and 
therefore  further  steps  would  be  needed  to  capture  the  full  value  of  the  Air 
Force’s  virtual  training  investment.  As  of  May  2012,  the  Air  Force  had  not 


1 Q 

Program  elements  are  primary  units  of  the  Future  Years  Defense  Program  programming 
and  budgeting  system. 

pn 

Secretary  of  the  Air  Force,  Office  of  Information  Dominance  and  Chief  Information 
Officer,  Warfighter  Systems  Integration,  Air  Force  Modeling  and  Simulation  Enterprise 
Management  Study  (Sept.  12,  201 1 ). 
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taken  any  additional  steps  to  develop  a  methodology  for  identifying  virtual 
training  costs.  Without  a  means  to  collect  or  calculate  its  virtual  training 
costs,  the  Air  Force  lacks  the  information  it  needs  to  make  informed 
investment  decisions  in  the  future  regarding  the  mix  of  live  and  virtual 
training.  Furthermore,  the  Air  Force  will  be  unable  to  determine  the 
potential  costs  associated  with  its  flying-hour  efficiency  initiative. 


Conclusions 


In  an  effort  to  achieve  greater  efficiencies  in  its  training  program  while 
maintaining  mission  readiness,  the  Air  Force  has  taken  various  steps  to 
emphasize  and  increase  the  use  of  virtual  training.  Among  other  things, 
the  Air  Force  has  implemented  various  initiatives  and  established 
organizations  intended  to  enhance  its  virtual  training  capabilities. 

However,  none  of  these  organizations  have  the  authority  necessary  to 
ensure  the  integration  of  the  Air  Force’s  virtual  training  efforts,  and 
oversight  remains  fragmented.  Further,  the  Air  Force  lacks  an 
overarching  organizational  framework  to  guide  its  current  virtual  training 
efforts  and  the  additional  investments  it  plans  to  make.  In  the  absence  of 
such  a  framework,  the  Air  Force  faces  challenges  in  managing  its  current 
inventory  of  virtual  training  systems  and  has  experienced  delays  and 
costs  that  stem  from  the  lack  of  interoperability  among  its  simulators  and 
networks,  resulting  in  workarounds  that  are  required  to  compensate  for 
these  limitations.  An  overarching  management  approach,  including  a 
single  entity  responsible  for  coordinating  and  integrating  all  virtual  training 
efforts,  as  well  as  a  comprehensive  strategy  that  aligns  individual  efforts 
with  goals  and  investment  priorities,  will  not  be  enough  if  decision  makers 
lack  visibility  over  the  potential  costs  of  virtual  training — especially  as  they 
consider  future  changes  to  the  mix  of  live  and  virtual  training.  Until  the  Air 
Force  has  a  methodology  to  consistently  collect  and  track  its  virtual 
training  costs  and  a  management  framework  to  coordinate  its  efforts,  it 
will  continue  to  face  challenges  to  planning  and  conducting  its  virtual 
training  and  informing  its  future  investment  decisions. 


Recommendations  for  develop  a  fully  integrated  management  approach  to  guide  virtual 

training  efforts  and  investments,  we  recommend  the  Secretary  of  Defense 
Executive  Action  direct  the  Secretary  of  the  Air  Force  to 

•  designate  an  entity  that  is  responsible  and  accountable  for  integrating 
all  of  the  Air  Force’s  virtual  training  efforts,  including  the  development 
and  enforcement  of  interoperability  standards  across  virtual  training 
systems,  and  investment  planning;  and 
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•  develop  an  overarching  strategy  to  align  goals  and  funding  for  virtual 
training  efforts  across  all  Air  Force  major  commands.  This  strategy 
should  at  a  minimum  contain  elements  such  as  results-oriented  goals, 
performance  measures,  and  a  determination  of  resources  needed  to 
achieve  stated  goals.  In  addition,  this  strategy  should  show  clear 
linkages  between  existing  and  planned  initiatives  and  goals. 

To  improve  decision  makers’  visibility  over  the  costs  related  to  virtual 
training,  we  recommend  that  the  Secretary  of  Defense  direct  the 
Secretary  of  the  Air  Force  to  develop  a  methodology  for  collecting  and 
tracking  cost  data  for  virtual  training  and  use  this  cost  data  to  help  inform 
future  decisions  regarding  the  mix  of  live  and  virtual  training. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  DOD  stated  that  it  concurred 
with  all  of  our  recommendations.  In  response  to  our  recommendation  to 
designate  an  entity  that  is  responsible  and  accountable  for  integrating  all 
of  the  Air  Force’s  virtual  training  efforts,  including  the  development  and 
enforcement  of  interoperability  standards  across  virtual  training  systems, 
and  investment  planning,  DOD  stated  that  the  Air  Force  has  taken  initial 
steps  to  designate  its  Headquarters,  Air  Force  office,  AF/A3/5 
(Operations,  Plans,  and  Requirements)  as  the  single  entity  responsible 
for  integrating  the  Air  Force’s  virtual  training  efforts. 

In  response  to  our  recommendation  to  develop  an  overarching  strategy  to 
align  goals  and  funding  for  virtual  training  efforts  across  all  Air  Force 
major  commands,  DOD  stated  that  the  Air  Force  is  developing  an 
overarching  strategy  and  policy  to  provide  a  fully  integrated  management 
approach  to  guide  its  Live  Virtual  Constructive-Operational  Training 
efforts  and  investments.  It  further  stated  that  operational  level  guidance 
will  be  provided  in  Air  Force  Instruction  36-2251,  Management  of  Air 
Force  Training  Systems  and  that  investment  guidance  to  link  virtual 
training  to  “Readiness”  was  provided  in  the  Air  Force’s  Fiscal  Year  2014 
Annual  Planning  and  Programming  Guidance  and  Program  Objective 
Memoranda  Preparation  Instructions. 

In  response  to  our  recommendation  to  develop  a  methodology  for 
collecting  and  tracking  cost  data  for  virtual  training  and  use  this  cost  data 
to  help  inform  future  decisions  regarding  the  mix  of  live  and  virtual 
training,  DOD  stated  that  the  Air  Force  is  taking  actions  to  improve 
visibility  related  to  virtual  training  to  inform  decisions  regarding  the  mix  of 
live  and  virtual  training.  It  also  stated  that  the  Air  Force  Instruction 
regarding  Management  of  Air  Force  Training  Systems  will  provide  major 
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commands  with  clear  guidance  to  employ  consistent  methods  to  collect 
and  measure  virtual  training  systems  data.  DOD  said  the  Aviation 
Resource  Management  System  will  be  enhanced  to  provide  the  capability 
to  capture  projected  and  executed  aircraft  virtual  training  and  cost  data  to 
provide  better  oversight  and  management  of  virtual  training  funding. 
Finally,  DOD  stated  that  the  Air  Force  is  developing  a  standard 
methodology  of  accounting  and  tracking  the  programming  and  execution 
of  program  funds  through  improved  visibility  into  cost  categories 
associated  with  Live  Virtual  Constructive-Operational  Training.  DOD’s 
comments  are  included  in  their  entirety  in  appendix  II.  DOD  also  provided 
a  number  of  technical  and  clarifying  comments,  which  we  have 
incorporated  where  appropriate. 


We  are  sending  copies  of  this  report  to  appropriate  congressional 
committees,  the  Secretary  of  Defense,  the  Secretary  of  the  Air  Force,  and 
the  Under  Secretary  of  Defense  for  Personnel  and  Readiness.  In  addition, 
this  report  will  be  available  at  no  charge  on  our  website  at 

http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact 
me  at  (202)  512-9619  or  pickups@gao.gov.  Contact  points  for  our  Offices 
of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last 
page  of  this  report.  GAO  staff  who  made  major  contributions  to  this  report 
are  listed  in  appendix  III. 


Sharon  L.  Pickup 
Director 

Defense  Capabilities  and  Management 
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Appendix  I:  Scope  and  Methodology 


To  address  our  objectives,  we  met  with  officials  from  the  Office  of  the 
Secretary  of  Defense,  Joint  Staff,  Office  of  the  Secretary  of  the  Air  Force, 
Headquarters  Air  Force,  and  several  Air  Force  major  commands.  Our 
review  focused  primarily  on  virtual  training  systems  for  manned  aircraft 
from  combat  air  forces,  mobility  air  forces,  and  special  operations  forces. 
Excluded  from  this  review  were  virtual  training  programs  for  unmanned 
aircraft,  space,  combat  support,  and  combat  service  support  systems.  To 
determine  how  the  Air  Force  determines  the  mix  of  live  and  virtual 
training,  we  obtained  and  analyzed  training  requirement  instructions  for 
combat,  mobility,  and  special  operations  aircraft  from  each  of  the  three 
lead  major  commands — Air  Mobility  Command,  Air  Combat  Command, 
and  Air  Force  Special  Operations  Command.  We  provided  a 
questionnaire  and  received  written  responses  from  the  major  commands 
on  the  mix  of  live  and  virtual  training  and  the  benefits,  limitations,  and 
challenges  of  virtual  training.  We  interviewed  officials  from  Air  Combat 
Command,  Air  Mobility  Command,  Air  Force  Special  Operations 
Command,  Air  National  Guard,  Air  Force  Reserve  Command,  U.S.  Air 
Forces  Europe,  and  Pacific  Air  Forces. 

To  determine  the  extent  to  which  the  Air  Force  has  developed  an 
overarching  framework  to  guide,  oversee,  and  integrate  its  virtual  training 
efforts,  we  analyzed  Air  Force  studies  on  virtual  training  technologies  and 
capabilities.  We  reviewed  relevant  Department  of  Defense  (DOD)  and 
Navy  training  guidance.  We  also  reviewed  our  ongoing  work  related  to 
Navy  virtual  training.  We  interviewed  officials  from  the  Office  of  the 
Secretary  of  the  Air  Force,  Headquarters  Air  Force,  the  Joint  Staff,  the 
Department  of  the  Navy,  Air  Force  Major  Commands,  the  four  primary 
centers  that  facilitate  distributed  mission  operations,  and  joint  training 
officials.  We  visited  Langley  Air  Force  Base,  Virginia,  to  observe  F-15  and 
F-22  simulator  operations  at  the  Mission  Training  Center.  We  also  visited 
the  Distributed  Mission  Operations  Center  at  Kirtland  Air  Force  Base, 

New  Mexico,  to  observe  a  distributed  training  event  and  the  center’s 
capabilities. 

To  determine  the  extent  to  which  the  Air  Force  considered  costs  related 
to  virtual  training  in  estimating  potential  savings  from  its  training  efficiency 
initiative,  we  obtained  and  analyzed  the  Air  Force  efficiency  calculation 
and  compared  it  with  cost  saving  estimating  best  practices.  We  analyzed 
Air  Force  budget  program  elements  related  to  virtual  training  and 
distributed  mission  operations.  We  also  analyzed  an  Air  Force  study  on 
the  cost  associated  with  Air  Force  modeling  and  simulation  efforts.  We 
interviewed  officials  from  the  Air  Force  Defense  Contracting  Management 
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Organization  and  Headquarters  Air  Force  budget  operations  office,  as 
well  as  officials  from  the  major  commands  listed  above. 

We  conducted  this  performance  audit  from  August  201 1  to  July  2012  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 

We  interviewed  officials,  and  where  appropriate  obtained  documentation, 
at  the  following  locations: 

•  Office  of  the  Secretary  of  Defense,  Office  of  the  Under  Secretary  of 
Defense  for  Personnel  and  Readiness 

•  Office  of  the  Under  Secretary  of  the  Air  Force,  Deputy  Chief 
Management  Officer 

•  Joint  Staff,  J7,  Joint  and  Coalition  Warfighting 

•  Headquarters  Air  Force,  Operations  Systems  Integration  (Live,  Virtual, 
and  Constructive-Operational  Training) 

•  Headquarters  Air  Force,  Operations  Systems  Integration  (Air  Force 
Agency  for  Modeling  and  Simulation) 

•  Headquarters  Air  Force,  Operations  Integration  and  Resources 

•  Headquarters  Air  Force,  Operations  Force  Management  (Force 
Integration) 

•  Headquarters  Air  Force,  Air  Force  Financial  Management  and 
Comptroller,  Budget  Operations  Office 

•  Office  of  the  Secretary  of  the  Air  Force,  Office  of  Security, 
Counterintelligence,  and  Special  Program  Oversight 

•  Air  Combat  Command,  Operations  and  Training 

•  Air  Mobility  Command,  Operations  and  Training 
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•  Air  Force  Special  Operations  Command,  Operations  and  Training 

•  Air  National  Guard,  Operations  and  Training 

•  Air  Force  Reserve  Command,  Operations  and  Training 

•  U.S.  Air  Forces  Europe,  Operations  and  Training 

•  Pacific  Air  Forces,  Operations  and  Training 

•  Air  Force  Material  Command,  Aeronautical  Systems  Center- 
Simulators  Division 

•  Air  Force  Space  Command,  Air  Force  Network  Integration  Center, 
Cross  Domain  Solutions  Office 

•  Distributed  Mission  Operations  Center 

•  Distributed  Training  Operations  Center 

•  Warrior  Preparation  Center 

•  Korean  Air  Simulation  Center 

•  Joint  Strike  Fighter  Program  Office 

•  Joint  Training  Integration  and  Evaluation  Center 

•  Navy  Naval  Air  Warfare  Center — Training  Systems  Division 
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of  Defense 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

4000  DEFENSE  PENTAGON 
WASHINGTON,  D.C.  20301-4000 


JUI  0  6  ?01? 

Ms.  Sharon  L.  Pickup 

Director,  Defense  Capabilities  and  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Ms.  Pickup, 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Report,  GAO- 1 2-727, 

"AIR  FORCE  TRAINING:  Actions  Needed  to  Better  Manage  and  Determine  Costs  of  Virtual 
Training  Efforts,”  dated  June  7,  2012  (GAO  Code  351655).  Thank  you  for  the  opportunity  to 
comment.  We  concur  with  the  recommendations.  Details  concerning  the  recommendations  are 
provided  in  the  enclosure  to  this  letter. 

Sincerely, 

Laura  J.  Junor 

Deputy  Assistant  Secretary  of  Defense 
Readiness 


PERSONNEL  AND 
READINESS 
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of  Defense 


GAO  DRAFT  REPORT  DATED  JUNE  7,  2012 
GAO-12-727  (GAO  CODE  351655) 

“AIR  FORCE  TRAINING:  ACTIONS  NEEDED  TO  BETTER  MANAGE  AND 
DETERMINE  COSTS  OF  VIRTUAL  TRAINING  EFFORTS” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  I:  To  develop  a  fully  integrated  management  approach  to  guide 
virtual  training  efforts  and  investments,  GAO  recommends  the  Secretary  of  Defense  direct  the 
Secretary  of  the  Air  Force  to  designate  an  entity  that  is  responsible  and  accountable  for 
integrating  all  of  the  Air  Force's  virtual  training  efforts,  including  the  development  and 
enforcement  of  interoperability  standards  across  virtual  training  systems,  and  investment 
planning. 

DoD  RESPONSE:  Concur.  The  Air  Force  has  taken  initial  steps  to  designate  a  single  entity 
to  develop  and  enforce  interoperability  standards  across  virtual  and  constructive  training 
communities.  AF/A3/5  (Operations,  Plans,  and  Requirements)  will  be  the  Air  Force’s  single 
entity  responsible  for  integrating  the  Air  Force’s  virtual  training  efforts. 

RECOMMENDATION  2:  To  develop  a  fully  integrated  management  approach  to  guide 
virtual  training  efforts  and  investments,  GAO  recommends  the  Secretary  of  Defense  direct  the 
Secretary  of  the  Air  Force  to  develop  an  overarching  strategy  to  align  goals  and  funding  for 
virtual  training  efforts  across  all  Air  Force  major  commands.  This  strategy  should  at  a  minimum 
contain  elements  such  as  results-oriented  goals,  performance  measures,  and  a  determination  of 
resources  needed  to  achieve  stated  goals.  In  addition,  this  strategy  should  show  clear  linkages 
between  existing  and  planned  initiatives  and  goals. 

DoD  RESPONSE:  Concur.  The  Air  Force  is  developing  overarching  strategy  and  policy  to 
provide  a  fully  integrated  management  approach  to  guide  Live  Virtual  Constructive-Operational 
Training  (LVC-OT)  efforts  and  investments.  Operational  level  guidance,  putting  the  Flight  Plan 
into  action,  will  be  provided  in  Air  Force  Instruction  36-2251,  Management  of  Air  Force 
Training  Systems.  In  addition,  LVC-OT  investment  guidance  was  provided  in  the  Air  Force’s 
FY14  Annual  Planning  and  Programming  Guidance  and  the  POM  Preparation  Instructions  to 
link  virtual  training  to  “Readiness.” 

RECOMMENDATION  3:  To  improve  decision  makers'  visibility  over  the  costs  related  to 
virtual  training,  GAO  recommends  that  the  Secretary  of  Defense  direct  the  Secretary  of  the  Air 
Force  to  develop  a  methodology  for  collecting  and  tracking  cost  data  for  virtual  training  and  use 
this  cost  data  to  help  inform  future  decisions  regarding  the  mix  of  live  and  virtual  training. 
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Appendix  II:  Comments  from  the  Department 
of  Defense 


DoD  RESPONSE:  Concur.  The  Air  Force  is  taking  actions  to  improve  visibility  related  to 
virtual  training  to  inform  decisions  regarding  the  mix  of  live  and  virtual  training.  The  review 
and  update  of  the  Air  Force  Instruction  regarding  “Management  of  Air  Force  Training  Systems” 
will  provide  MAJCOMs  with  clear  guidance  to  employ  consistent  methods  to  collect  and 
measure  virtual  training  systems  data.  In  addition,  the  Aviation  Resource  Management  System 
will  be  enhanced  to  provide  the  capability  to  capture  projected  and  executed  aircraft  virtual 
training  and  cost  data  to  provide  better  oversight  and  management  of  virtual  training  funding. 
Currently,  virtual  and  constructive  training  are  funded  through  35+  program  elements.  The  Air 
Force  is  developing  a  standard  methodology  of  accounting  and  tracking  the  programming  and 
execution  of  the  program  funds  through  improved  visibility  into  cost  categories  associated  with 
LVC-OT. 
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Appendix  III:  GAO  Contact  and  Staff 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  website  (www.gao.gov).  Each  weekday  afternoon, 
GAO  posts  on  its  website  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  website, 

http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 
Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512- 
4400,  U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room 
7125,  Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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